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Abstract 

This report summarizes the activities of the Haystack Correlator during 2012. Highlights include 
finding a solution to the DiFX InfiniBand timeout problem and other DiFX software development, 
conducting a DBE comparison test following the First International VLBI Technology Workshop, 
conducting a Mark IV and DiFX correlator comparison, more broadband delay experiments, more u- 
VLBI Galactic Center observations, and conversion of RDV session processing to the Mark IV /HOPS 
path. Non-real-time e-VLBI transfers and engineering support of other correlators continued. 


1. Introduction 

The Mark IV and DiFX VLBI correlators of the MIT Haystack Observatory, located in West- 
ford, Massachusetts, are supported by the NASA Space Geodesy Program and the National Science 
Foundation. They are dedicated mainly to the pursuits of the IVS, with a smaller fraction of time 
allocated to processing radio astronomy observations for the Ultra High Sensitivity VLBI (u-VLBI) 
project. The Haystack correlators serve as development systems for testing new correlation modes, 
for hardware improvements such as the Mark 6 system, and in the case of the Mark IV, for diagnos- 
ing correlator problems encountered at Haystack and at the identical correlator at the U.S. Naval 
Observatory. This flexibility is made possible by the presence on-site of the team that designed 
the Mark IV correlator hardware and software. Some software support is provided to the Max 
Planck Institute for Radioastronomy in Bonn, Germany, for DiFX processing of IVS sessions. 

2. Summary of Activities 

2.1. DiFX Cluster Developments 

Efforts to diagnose InfiniBand fabric timeout errors which manifested themselves at high record 
rates (1 Gb/sec and above) have come to fruition through an understanding that the particular 
model of InfiniBand cards installed in the Mark 5 playback units were the cause. New cards were 
procured and tested, and they appear to mitigate the problem. Cards were ordered and installed 
in all the capable Mark 5 units. Various other DiFX related debugging projects were conducted, 
and problems were corrected. 

2.2. DiFX Software Support 

Initially mysterious problems with multiband delay (mbd) scatter in many DiFX correlated 
sessions from Bonn led to an investigation at Haystack. Data for session R1543 were transferred 
from Bonn to Haystack and correlated on the Haystack Mark IV for comparison. This revealed a 
problem with time registration of delay polynomials in the difx2mark4 conversion software. The 
problem was corrected but not before ~ 120 sessions were affected. Fortunately, the delays can 
be corrected in the databases; as of the end of 2012, NASA Goddard and BKG had corrected 
approximately half of the databases. 
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2.3. DiFX-Mark IV Correlator Comparison of Experiment R1543 

As a by-product of the diagnosis of the difx2mark4 problem, a comparison between DiFX 
and the Mark IV correlator was conducted. Results show that mbd differences compare favor- 
ably between all combinations of crossings. The full analysis of this comparison be found at 
http:// cira. ivec. org/dokuwiki/ doku.php/difx/ difx2mark4 

2.4. Broadband Delay 

Major broadband delay tests were conducted in January, May, and October 2012. January had 
a plethora of tests, and May and October featured six- hour “geodetic quality” schedules being run 
at the GGAO 12-m and Westford antennas to fully test the VLBI2010 system. Various other tests 
were conducted over the year, including overlapping bands, zoom mode (Mark IV vs. broadband), 
a source transit experiment, a DBEl/Mark 5B+ vs. RDBE/Mark 5C comparison, and others. 

2.5. DBE Comparison Test 

Following the First International VLBI Technology Workshop held at Haystack in Oc- 
tober, a DBE Compatibility Testing Workshop was hosted by Haystack staff. DBEs from 
Haystack, China, Japan, and Europe were compared to test interoperability. All units partic- 
ipating in the workshop were successfully compared. A report of the results can be found at 
http://www.haystack.mit.edu/workshop/ivtw/index.html. As a by-product of this test Haystack 
provided the DiFX correlation setup to Seshan in order for them to duplicate the Haystack results. 
This can be used for future debugging of their DBE on their DiFX installation. 

2.6. Galactic Center Observations 

Further u-VLBI observations of the Galactic Center with dual polarization at all sites were 
recorded and correlated in 2012. Also, results were published for the ground-breaking observations 
made in 2011 of the center of the galaxy M87 that spatially resolved the base of the jet in that 
source. Fringe searching to the MPI/Onsala/ESO sponsored APEX antenna in Atacama, Chile, 
was successful. These searches were conducted in Bonn, but searches for the participating U.S. 
stations were done at Haystack. In the near future other new antennas will be tested and added 
to the u-VLBI array, which will greatly increase its resolution. 

2.7. RDV Fringe-fitting 

Test fringe-fitting of the RDV sessions correlated on the NRAO DiFX correlator described last 
year showed favorable results compared to the traditional NRAO AIPS package. Thus, processing 
them through the Mark IV/HOPS path has become the routine production process. A summary 
of this work can be found at http://www.oan.es/gm2012/pdf/postemid-117.pdf 

2.8. e-VLBI 

Non-real-time transfers have continued. Data from nineteen experiments were transferred to 
Haystack this year from seventeen stations (seven in Japan, four in Western Europe, two in Aus- 
tralia, two in South America, one in Crimea, and one in South Africa): Kashimall, Koganei, 
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Tsukuba, Chichijima, Ishigaki, Aira, Mizusawa, Onsala, Ny-Alesund, Wetzell, Noto, Hobart, Yara- 
gadee, Fortaleza, Tigo (via Bonn), Crimea (via Bonn) and HartRAO. e-VLBI transfers have sig- 
nificantly increased this year due to an upgrade of Haystack’s connectivity to the Internet which 
has enabled data transfer rates up to 1.4 Gb/sec. 

2.9. Experiments Correlated 

Production processing on the Mark IV correlator continues amidst all the DiFX development. 
In 2012, thirty-four geodetic VLBI experiments were processed, at least in part, on the Haystack 
Mark IV correlator, including ten R&Ds, six T2s, one AUST, and the aforementioned R1543 exper- 
iment. The remaining 16 were various tests. The other test experiments included the broadband 
experiments and fringe tests and an assortment of other projects, some of which were mentioned 
in the summary above. As usual, smaller tests were not included in the above count because they 
were too small to warrant individual experiment numbers. 

2.10. Current/Future Hardware and Capabilities 

As of the end of 2012 the Mark IV correlator was comprised of seven Mark 5A units, seven 
station units, seven Mark 5B units (DOMs) with their associated correlator interface boards (CIBs), 
16 operational correlator boards, two crates, and miscellaneous other support hardware. We have 
the capacity to simultaneously process all baselines for 11 stations in the standard geodetic modes, 
provided the aggregate recordings match the above hardware matrix. Note that all experiments 
up to 15 stations have been done in one pass due to the ability to share playback units between 
stations which do not co-observe. Six of the playback units are accessible to the six-server (12 
cores each) DiFX cluster. 

In 2013 we hope to transition to the software correlator, only keeping the hardware correlator 
alive in support of USNO until their transition to a software correlator, which is expected in late 
2013. 

3. Staff 

Staff who participated in aspects of Mark IV, DiFX, Mark 5/6, and e-VLBI development and 
operations include: 

3.1. Software Development Team 

• John Ball - Mark 5A/5B; e-VLBI. 

• Roger Cappallo - real-time correlator software and troubleshooting; system integration; post 
processing; Mark 5B/5C/6; Linux conversion; e-VLBI; DiFX correlator development. 

• Geoff Crew - DiFX correlator development, post processing software; Mark 6. 

• Kevin Dudevoir - correlation; maintenance/support; Mark 5A/5B/5C; e-VLBI; Linux 
conversion; correlator software and build system development; computer system sup- 
port/development; DiFX correlator development. 

• Jason SooHoo - e-VLBI; Mark 5A/5B/5C/6; computer system support. 
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• Chester Ruszczyk - e-VLBI; Mark 5A/5B/5C. 

• Alan Whitney - system architecture; Mark 5A/5B/5C/6; e-VLBI . 

3.2. Operations Team: 

• Peter Bolis - correlator maintenance. 

• Alex Burns - playback drive maintenance; Mark 5 installation and maintenance; general 
technical support; replacement for Dave Fields (see below). 

• Brian Corey - experiment correlation oversight; station evaluation; technique development. 

• Dave Fields - playback drive maintenance; Mark 5 installation and maintenance; general 
technical support; retired in December 2012. 

• Glenn Millson - correlator operator. 

• Arthur Niell - technique development. 

• Don Sousa - correlator operator; experiment setup; tape library and shipping. 

• Mike Titus - correlator operations oversight; experiment setup; computer services; software 
and hardware testing. 

• Ken Wilson - correlator maintenance; playback drive maintenance; general technical support. 

4. Conclusion/Outlook 

A full transition to the DiFX software correlator is expected in early 2013. Operational testing 
of the complete VLBI2010 system is expected to start in early 2013. Testing and implementation 
of new digital back ends and recording systems will continue. 
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